Image data classification method, device and system

ABSTRACT

The present disclosure relates to an image data classification method, device and system, and relates to the field of computer technology. The method includes: inputting test image data into a neural network model trained by using an original training sample set for classification, and determining an image type to which the test image data belongs and a membership probability of the image data belonging to the image type; establishing an easy-to-classify data set, according to test image data with a membership probability greater than a first threshold; adding test image data in the easy-to-classify data set that has a classification accuracy rate less than or equal to a second threshold and a correct classification result to the original training sample set to generate an augmented training sample set; and using the augmented training sample set to train the neural network model so as to determine an image class

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is based on and claims priority to China PatentApplication No. 202010996020.4 filed on Sep. 21, 2020, the disclosure ofwhich is incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to the field of computer technology, andin particular to an image data classification method, an image dataclassification device, an image data classification system, and anon-volatile computer-readable storage medium.

BACKGROUND

With the emergence of deep neural networks (DNN), the image processingtechnology has been greatly developed. Owing to faster speed and higherprecision, deep neural networks have been more widely applied in variousfields. For example, in practical applications such as medicine, remotesensing, face recognition, and automatic driving, a large number ofimages may be classified and processed by the image processingtechnology based on deep neural networks.

In related technologies, based on the classification criteria of servicescenarios, training samples of various images are collected to trainneural network models; images are classified by the trained neuralnetwork models.

SUMMARY

According to some embodiments of the present disclosure, an image dataclassification method is provided. The method comprises the steps of:inputting test image data into a neural network model trained by usingan original training sample set for classification, and determining animage type to which the test image data belongs and a membershipprobability of the image data belonging to the image type; establishingan easy-to-classify data set, according to test image data with amembership probability greater than a first threshold; adding test imagedata in the easy-to-classify data set that has a classification accuracyrate less than or equal to a second threshold and a correctclassification result to the original training sample set so as togenerate an augmented training sample set; and using the augmentedtraining sample set to train the neural network model so as to determinean image classification model.

In some embodiments, the classification method further comprises: addingtest image data with a classification accuracy rate greater than thesecond threshold in the easy-to-classify data set to the training sampleset so as to generate the augmented training sample set.

In some embodiments, the classification method further comprises:inputting the test image data into the neural network model trained bythe augmented training sample set for classification; processing theaugmented training sample set based on whether test image data that isincorrectly classified this time pertains to the easy-to-classify dataset; and using the augmented training sample set after processing totrain the neural network model again so as to update the imageclassification model.

In some embodiments, the classification method further comprises: usingtest image data that is correctly classified this time to augment theaugmented training sample set again so as to train the neural networkmodel again.

In some embodiments, the step of inputting each of the test image datainto the neural network model trained by the augmented training sampleset for classification comprises: inputting a first data set and asecond data set of the test image data into the neural network modeltrained by the augmented training sample set for classification, whereinthe first data set comprises test image data in the easy-to-classifydata set that has a classification accuracy rate less than or equal tothe second threshold and an incorrect classification result, and thesecond data set comprises test image data in a difficult-to-classifydata set, wherein the difficult-to-classify data set is determined basedon an image type to which test image data with a membership probabilityless than or equal to the first threshold belongs.

In some embodiments, the step of processing the augmented trainingsample set based on whether the test image data that is incorrectlyclassified this time pertains to the easy-to-classify data setcomprises: determining an incorrectly marked samples required to befiltered in the augmented training sample set based on the test imagedata that is incorrectly classified in a case that the test image datathat is incorrectly classified belongs to the first data set.

In some embodiments, the step of processing the augmented trainingsample set based on whether the test image data that is incorrectlyclassified this time pertains to the easy-to-classify data setcomprises: obtaining sample data of an image type of the test image datathat is incorrectly classified this time to augment the augmentedtraining sample set again, in the case where the test image data that isincorrectly classified pertains to the difficult-to-classify data set,wherein the difficult-to-classify data set is determined based on animage type to which test image data with a membership probability lessthan or equal to the first threshold belongs.

In some embodiments, the step of obtaining sample data of an image typeof the test image data that is incorrectly classified this time toaugment the augmented training sample set again comprises: performingdata enhancement processing on the test image data that is incorrectlyclassified this time, obtaining the sample data of the image type of thetest image data that is incorrectly classified this time, and augmentingthe augmented training sample set again.

In some embodiments, the classification method further comprises:calculating a classification accuracy rate of the neural network modeltrained again by using the augmented training sample set afterprocessing; repeating at least one of the following steps until theclassification accuracy rate of the neural network model is greater thanthe third threshold in a case where the classification accuracy rate ofthe neural network model is less than or equal to the third threshold:augmenting test image data in the easy-to-classify data set that has aclassification accuracy less than or equal to the second threshold and acorrect classification result to the training sample set, and trainingthe neural network model by using the training sample set augmented; orprocessing the training sample set based on whether the test image datathat is incorrectly classified this time belongs to the easy-to-classifydata set, and training the neural network model using the processedtraining sample set.

In some embodiments, the classification method further comprises: usinga linear adjustment factor and an exponential adjustment factor providedfor a membership probability to process a focus loss function with themembership probability as a variable, and determining an improved lossfunction to train the neural network mode, wherein the linear adjustmentfactor and the exponential adjustment factor are configured such thatvalue of the improved loss function corresponding to the membershipprobability is greater than that of the focal loss function in the casewhere any membership probability is less than a membership probabilityof the intersection point of the focal loss function and the improvedloss function, and the value of the improved loss function correspondingto the membership probability is less than that of the focus lossfunction in a case where any membership probability is greater than amembership probability of the intersection point.

In some embodiments, the improved loss function is determined accordingto (1−ŷ+ε)^(γ), wherein ŷ is the membership probability, ε is the linearadjustment factor, and γ is the exponential adjustment factor.

In some embodiments, the classification method further comprises:inputting image data into the image classification model and determiningan image type to which the test image data belongs.

In some embodiments, the image data is production line image data ofproduction industry, and the image type is a product defect type of theproduction line image data.

In some embodiments, the classification method further comprises:inputting image data into the image classification model and marking theimage data based on a classification result.

In some embodiments, the neural network model is Visual Geometry GroupNetwork model.

In some embodiments, the original training sample set is obtained bycapturing product images during production process.

In some embodiments, the classification accuracy rate is aclassification accuracy rate of an image type in the easy-to-classifydata set calculated by using multiple accuracy rate detection modules.

According to yet other embodiments of the present disclosure, an imagedata classification system is provided. The system comprises: an imagedata classification device according to any one of the above-describedembodiments; and an image sensor for obtaining image data.

According to still other embodiments of the present disclosure, an imagedata classification device is provided. The device comprises: a memory;and a processor coupled to the memory, wherein the processor isconfigured to implement the image data classification method accordingto any one of the above-described embodiments based on instructionsstored in the memory. According to still other embodiments of thepresent disclosure, a non-volatile computer-readable storage medium isprovided. The medium has a computer program stored thereon, which whenexecuted by a processor implements the image data classification methodaccording to any one of the above-described embodiments.

BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS

The accompanying drawings which constitute part of this specification,illustrate the exemplary embodiments of the present disclosure, andtogether with this specification, serve to explain the principles of thepresent disclosure.

The present disclosure may be more clearly understood from the followingdetailed description with reference to the accompanying drawings.

FIG. 1 shows a flowchart in some embodiments of the image dataclassification method according to the present disclosure;

FIG. 2 shows a flowchart in some embodiments of step 120 and step 130 inFIG. 1;

FIG. 3 shows a flowchart in some embodiments of step 140 in FIG. 1;

FIG. 4 shows a schematic view in some embodiments of the image dataclassification method according to the present disclosure;

FIG. 5 shows a block diagram in some embodiments of the image dataclassification device according to the present disclosure;

FIG. 6 shows a block diagram of other embodiments of the image dataclassification device according to the present disclosure;

FIG. 7 shows a block diagram in still other embodiments of the imagedata classification device according to the present disclosure;

FIG. 8 shows a block diagram in some embodiments of the image dataclassification system of the present disclosure.

DETAILED DESCRIPTION

Various exemplary embodiments of the present disclosure will now bedescribed in detail with reference to the accompanying drawings. Itshould be noted that: unless additionally specified, the relativearrangements, numerical expressions and numerical values of thecomponents and steps expounded in these examples do not limit the scopeof the present disclosure.

At the same time, it should be understood that, for ease of description,the dimensions of various parts shown in the accompanying drawings arenot drawn according to actual proportional relations.

The following descriptions of at least one exemplary embodiment whichare in fact merely illustrative, shall by no means serve as anydelimitation on the present disclosure as well as its application oruse.

The techniques, methods, and apparatuses known to those of ordinaryskill in the relevant art might not be discussed in detail. However, thetechniques, methods, and apparatuses shall be considered as a part ofthe granted description where appropriate.

Among all the examples shown and discussed here, any specific valueshall be construed as being merely exemplary, rather than as beingrestrictive. Thus, other examples in the exemplary embodiments may havedifferent values.

It is to be noted that: similar reference signs and letters presentsimilar items in the following accompanying drawings, and therefore,once an item is defined in one accompanying drawing, it is necessary tomake further discussion on the same in the subsequent accompanyingdrawings.

The inventors of the present disclosure have found that theabove-described related technologies are present with the followingproblems: since the classification criteria of various service scenariosare complicated, the samples of various image types are unevenlydistributed, thereby resulting in low classification accuracy of theneural network model.

In view of this, the present disclosure provides a technical solution ofimage data classification, which is capable of augmenting the number ofsamples of image types with low classification accuracy, therebyimproving the classification accuracy of the neural network.

In some embodiments, by capturing the product images during a productionprocess (for example, a screen production process) and reviewing theproduct images by an engineer, it is possible to determine the type ofdefect and solve a damaged product in time. In this way, it is possibleto control the product quality and detect a product defect, therebyimproving the probability of qualified product.

In order to solve the technical problem of slow review speed caused bythe huge number of product images, for example, a deep neural networkmodel may be used to classify a large number of product images.

However, on the one hand, since the image sample data for traininginvolves high noise, and the image types are not clearly distinguished,it is likely to cause an increased probability of misclassification ofthe image sample data. Moreover, some image types are similar anddifficult to be distinguished, which may also cause an increasedprobability of misclassification of the image sample data.

In this way, it is possible to result in reduced accuracy of the imageprocessing technology based on deep neural networks, so that it isimpossible to effectively recognize the type of a product defect.

On the other hand, since the image sample data of various types isunevenly distributed, during the process of actually collecting theimage sample data, it is possible that the image sample data of acertain image type is missing or in a small amount. In other words,different product defects have different probabilities of occurrence,thereby resulting in a tailed property presented in the datadistribution of different image types.

Moreover, since the classification criteria in actual practices arecomplicated, it is likely that there are circumstances where some imagetypes contain multiple modes. For example, “oil stain” as an image typemay comprise multiple modes such as “light oil stain shaped” and“circular oil stain”. In this way, it is also possible that there arecircumstances where multiple modes are unevenly distributed even in thesame image type.

In this way, the image sample data is unevenly distributed, which mayresult in that the neural network model cannot recognize part of theimage types.

On the other hand, the classification data set generated afterclassification of the neural network model has a very large data scale,it takes a very long period to complete the data cleaning only manually,and it is impossible to effectively locate a problem present in theclassification data set.

In response to the above-described technical problem, the presentdisclosure provides an improved solution of a loss function, so that thetrained neural network model may better recognize and distinguish theimage types with a small amount of sample data. In addition, a technicalsolution of data cleaning is provided to effectively filter out thenoise data in the data set and reduce the workload of data review.Moreover, a mechanism for quickly locating the problem data is provided,which may effectively find out data in the data set that is difficult tobe classified, and provide a basis for the next round of data cleaning.For example, the technical solution of the present disclosure may berealized by the following embodiments.

FIG. 1 shows a flowchart in some embodiments of the image dataclassification method according to the present disclosure.

As shown in FIG. 1, the method comprises: step 110 of classifying testdata; step 120 of establishing an easy-to-classify data set; step 130 ofaugmenting training samples; and step 140 of training an imageclassification model.

In step 110, each test image data is input to the neural network modeltrained with an original training sample set for classification, so asto determine an image type to which each image data belongs and amembership probability of the image data belonging to the image type.For example, each image data is the production line image data of theproduction industry, and the image type is the product defect type inthe production line image data. The neural network model may be a VGG(Visual Geometry Group Network) model. The test image data is unlabeledimage data, which may be marked by the neural network model and theclassification review processing.

For example, the data amount of the test image data is far greater thanthat of the original training sample set. Therefore, the test image datais classified by the neural network model so that it is possible toimprove the efficiency of data labeling.

In some embodiments, the original training sample set may be obtained bycapturing the product images during the production process. For example,several sample image data (for example, 300 to 500) may be screened fromeach image type contained in the original training sample set fortraining the neural network model.

In some embodiments, the neural network model may be trained by using afocal loss function. For example, the linear adjustment factor and theexponential adjustment factor provided for the membership probabilitymay be used to process the focus loss function with the membershipprobability as a variable, and determine an improved loss function fortraining the neural network model.

The linear adjustment factor and the exponential adjustment factor areconfigured such that: in the case where any membership probability isless than the membership probability of an intersection point of thefocal loss function and the improved loss function, the value of theimproved loss function corresponding to the membership probability isgreater than that of the focal loss function; in the case where anymembership probability is greater than the membership probability of theintersection point, the value of the improved loss functioncorresponding to the membership probability is less than that of thefocal loss function.

In other words, the linear adjustment factor and the exponentialadjustment factor are configured such that the focus loss function has asteeper curve slope. Thus, the loss value of the loss function for datawith a small membership probability (less than the membershipprobability of the intersection point) is increased; and the loss valuefor data with a large membership probability (greater than theintersection point) is reduced.

For example, the improved loss function is determined based on(1−ŷ+ε)^(γ), where ŷ is the membership probability, ε is the linearadjustment factor (if the value is 0.15), and γ is the exponentialadjustment factor (if the value is 4). The improved focal loss functionis as follows:

FL=α(1−ŷ+ε)^(γ) ·y log(ŷ)

α is the balance factor (for example, the value is 0.25), which mayreduce the influence caused by the uneven number distribution among theimage types; y is for indicating whether the current image belongs tothe current image type (for example, the value of y is 0 or 1); γ maysmoothly adjust the loss value of the image based on the classificationdifficulty of the current image; ε may enhance the model's ability tolearn difficult-to-classify data and reduce the sensitivity to theclassified data.

In this way, the improved focus loss function is applied to the losslayer in the network structure, so that it is possible to reduce theloss value of easy-to-classify data and increase the loss value ofdifficult-to-classify data during the model training process. Therefore,the improved focus loss function allows the neural network model to havea stronger learning ability for defect data with a small number ofsamples and a special shape.

After the neural network model capable of classifying the images istrained, the model may be used to infer the data to be cleaned in thetraining sample set.

In step 120, the test image data with the membership probability greaterthan the first threshold is divided into the easy-to-classify data set.

In step 130, the test image data in the easy-to-classify data set whichhas a classification accuracy rate less than or equal to the secondthreshold and a correct classification result is added to the originaltraining sample set to generate an augmented training sample set.

For example, a plurality of accuracy rate detection modules (which maybe a plurality of engineers, a plurality of machine learning models, andthe like) may be used to calculate the classification accuracy rate ofeach image type in the easy-to-classify data set; to determine whetherthe classification accuracy rate of the image type to which each imagedata in the easy-to-classify data set belongs is less than or equal tothe second threshold; and to determine whether the presentclassification result of each image data in the easy-to-classify dataset is correct by review processing.

In some embodiments, the test image data with a classification accuracyrate greater than the second threshold in the easy-to-classify data setis added to the training sample set to generate an augmented trainingsample set.

In the above-described embodiments, based on the classification resultof each test data by the neural network model, the data belonging to theeasy-to-classify image type with low classification accuracy and correctclassification is screened from each test data, for augmenting thetraining sample. In this way, the number of samples of the image typewith low classification accuracy may be augmented, thereby improving theclassification accuracy of the neural network.

In some embodiments, step 120 and step 130 may be implemented by theembodiment in FIG. 2.

FIG. 2 shows a flowchart in some embodiments of step 120 and step 130 inFIG. 1.

As shown in FIG. 2, the step 120 comprises: step 1210 of obtaining theclassification result; step 1220 of determining whether the degree ofmembership is greater than the first threshold; and step 1230 ofestablishing the second data set. The step 130 comprises: step 1310 ofdetermining whether the accuracy of the easy-to-classify set is greaterthan the second threshold; step 1320 of determining whether theclassification is correct; step 1330 of augmenting the training sample;and step 1340 of establishing the first data set.

In step 1210, after each test image data is input into the neuralnetwork model, the image type (for example, the product defect type) ofeach test image data and its membership probability may be obtained.

In step 1220, it is determined whether the membership probability isgreater than the first threshold. In the case where it is greater thanthe first threshold, step 1310 is performed; in the case where it isless than or equal to the first threshold, step 1230 is performed.

For example, if the membership probability is greater than the firstthreshold (for example, 0.8), it is determined that the correspondingimage data passes the model test, may be favorably recognized by themodel, and determined as the easy-to-classify data; if the membershipprobability is less than or equal to the first threshold, It isdetermined that the corresponding image data cannot be favorablyrecognized by the model, and determined as difficult-to-classify data.The membership probability is the probability that the image databelongs to the image type.

In step 1230, the corresponding image data is divided into the seconddata set. The second data set contains the difficult-to-classify data.

In some embodiments, the first data set comprises test image data in theeasy-to-classify data set which has a classification accuracy rate lessthan or equal to the second threshold and an incorrect classificationresult. The second data set comprises test image data pertaining to thedifficult-to-classify data set. The difficult-to-classify data set isdetermined based on the image type to which the test image data with themembership probability less than or equal to the first thresholdbelongs.

In step 1310, it is determined whether the classification accuracy rateof the image type corresponding to each image data in theeasy-to-classify data set is greater than the second threshold. In thecase where the classification accuracy rate is greater than the secondthreshold, step 1330 is performed; in the case where the classificationaccuracy rate is less than or equal to the second threshold, step 1320is performed.

In some embodiments, the sample image data of total images is randomlyselected from each image type contained in the easy-to-classify dataset, and N accuracy detection modules (which may be N engineers, Nmachine learning models, and the like) are used to performclassification and review processing. The accuracy detection module isonly configured to detect whether the classification result is correct,but not to correct the classification result involving an incorrectclassification.

For example, for the class_k image type, total_(operator_i) image datais extracted and pushed to a plurality of accuracy detection modules fordetection. The accuracy detection module operator_i determines whetherthe pushed image data is class_k, and screens out the image data that isincorrectly classified. After the screening of the image data of thistype, the remaining data amount is residual_(operatir_i).

The classification accuracy of class_k judged by operator_i is:

${score}_{{operator}\_ i} = \frac{{residual}_{{operator}\_ i}}{{total}_{{operator}\_ i}}$

The average accuracy rate of all accuracy rate detection modules forthis type of data is calculated as the classification accuracy rate ofthis type of data:

${Quality}_{{class}\_ k} = \frac{\sum\limits_{i = 1}^{i = N}\;{score}_{{operator}\_ i}}{N}$

In some embodiments, the image data corresponding to the image type witha classification accuracy greater than the second threshold may bedivided into a high-quality image data set, so as to augment thetraining samples.

In some embodiments, the image data corresponding to the image type witha classification accuracy less than the second threshold may beclassified into a low-quality image data set, and determine whether toaugment the image data to the training samples by further determiningwhether the classification of the image data is correct. For example,for an image type with a classification accuracy rate of less than 0.9,several images (for example, 1000 to 2000) may be selected from thecorresponding image types in the easy-to-classify data set and stored inthe low-quality image data set.

In this way, the training sample set may be cleaned based on the imagequality, thereby constructing a new training sample set.

In step 1320, it is determined whether each image data in theeasy-to-classify data set is correctly classified. In the case of acorrect classification, step 1330 is performed; in the case of anincorrect classification, step 1340 is performed.

In step 1330, the image data in the low-quality data set that iscorrectly classified is augmented to the training sample set. The imagedata in the high-quality image data set may also be augmented to thetraining sample set.

In step 1340, the corresponding image data is divided into the firstdata set. The first data set comprises image data in theeasy-to-classify data set that belongs to the low-quality data set andis incorrectly classified, that is, the data that has been misclassifiedonce.

In some embodiments, the obtained low-quality data set is pushed to theaccuracy detection module for review. The accuracy detection modulescreens the image data that is incorrectly classified in the low-qualitydata set; determines the image data that is incorrectly classified asthe data that has been misclassified once, and augments the image datathat is correctly classified to the training sample set.

After the augmented training sample set is obtained, and the first dataset and the second data set are divided, image classification may beperformed by the remaining steps in FIG. 1.

In step 140, the neural network model is trained by the augmentedtraining sample set so as to determine the image classification model.

In some embodiments, step 140 may be implemented by the embodiment inFIG. 3.

FIG. 3 shows a flowchart in some embodiments of step 140 in FIG. 1.

As shown in FIG. 3, step 140 comprises: step 1410 of classifying thetest data again; step 1420 of determining whether it pertains to theeasy-to-classify data set; step 1430 of determining the training samplerequired to be augmented based on the incorrect classification data;step 1440 of determining the training samples required to be filteredbased on the incorrect classification data; and step 1450 of trainingthe neural network model again.

In step 1410, each test image data is input into the neural networkmodel trained by the augmented training sample set for classification.

In some embodiments, the first data set and the second data set in eachtest image data are input into the trained neural network model forclassification.

For example, the collective neural network model is updated by theaugmented training sample set (for example, trained by the improved lossfunction). The first data set comprising the data that has beenmisclassified once and the second data set comprising thedifficult-to-classify data are input into the updated neural networkmodel for classification.

In some embodiments, in this classification process, the test databelonging to the easy-to-classify data set and having a classificationaccuracy rate greater than the second threshold is divided into thethird data set. The test image data in the third data set is input intothe trained neural network model for performing classificationprocessing again. Based on whether the two classification results areconsistent, it is determined whether the test image data in the thirddata set is correctly classified by the neural network model.

For example, in the case where the two classification results areconsistent, it is determined that the classification is correct; the twoclassification results are inconsistent, and it is determined that theclassification is incorrect. In this way, it is possible to quicklydetermine whether the marking result of the image data in the third dataset marked by the neural network model is correct.

In step 1420, it is determined whether the test image data that isincorrectly classified this time belongs to the easy-to-classify dataset. In the case where the test image data belongs to theeasy-to-classify data set, step 1430 is performed; in the case where thetest image data pertains to the easy-to-classify data set, step 1440 isperformed.

In some embodiments, this classification result is pushed to theaccuracy detection module for review to determine whether thisclassification result is correct; the image data that is incorrectlyclassified again is screened; the image data that is incorrectlyclassified again is collected as the data that has been misclassifiedtwice.

In step 1430, the test image data that is incorrectly classified isderived from the difficult-to-classify data set, and the image type ofthe sample data that is required to be augmented again to augment thetraining sample set is determined based on the image type of the testimage data that is incorrectly classified this time.

In some embodiments, if the data that has been misclassified twicebelongs to the difficult-to-classify data set, it is indicated that thecurrent neural network model cannot recognize these image data. Theseimage data may belong to a new image type (for example, a new type ofproduct defect or a product with a defect that is difficult to berecognized).

Based on these image data, it is possible to quickly understand the newand abnormal product defects in actual practices, and further collectthe image data of similar types from the technical and service levels.

For example, from the technical level, the defect of this type may besimulated by the data enhancement technology to generate the imagesample data of the corresponding image type for augmenting the trainingsample set.

For example, from the practical level, it is possible to focus oncollecting the image data for the defect of this type in the productionwork, and further augment the training sample of this type.

In step 1440, the incorrectly marked samples required to be filtered inthe augmented training sample set are determined based on the test imagedata that is incorrectly classified. For example, in this case, the testimage data that is incorrectly classified belongs to the first data set,and the incorrectly marked samples required to be filtered may bedetermined based on the image type of the test data that is incorrectlyclassified.

In some embodiments, the data that has been misclassified twice isderived from the data that has been misclassified once, and these imagedata might be caused by the misclassified data (the samples that areincorrectly marked) in the original training sample set.

By these image data, it is possible to quickly understand the errors inthe delayed sample set so as to perform processing by data cleaning.

In step 1450, the neural network model is trained again by using theaugmented training sample set after processing so as to update the imageclassification model.

In some embodiments, the augmented training sample set after processingmay be used to train other neural network models for image dataclassification.

In some embodiments, the classification accuracy of the neural networkmodel trained again by using the augmented training sample set afterprocessing is calculated. The steps in FIGS. 2 and 3 may be repeatedlyperformed until the classification accuracy rate is greater than thethird threshold. In each repeated process, the parameters of the neuralnetwork model with the same network framework are updated.

In some embodiments, the augmented training sample set may be processedby at least one of the following processing methods: processing theaugmented training sample set based on whether the test image data thatis incorrectly classified belongs to the easy-to-classify data set; oraugmenting the augmented training sample set again by using the testimage data that is correctly classified this time.

For example, at least one of the following steps is repeated until theclassification accuracy rate is greater than the third threshold: thetest image data in the easy-to-classify data set that has aclassification accuracy rate less than or equal to the second thresholdand a correct classification result is used to augment the trainingsample; the training sample is processed based on whether the test imagedata that is incorrectly classified this time belongs to theeasy-to-classify data set.

In some embodiments, each image data is input into the imageclassification model to determine the image type to which each imagedata belongs.

For example, each image data is input into an image classificationmodel, and each image data is marked based on the classification result.In this way, it is possible to efficiently mark the image data, andlocate the previous marking errors.

In the above-described embodiments, the augmented training sample setthat has a larger scale and is more accurate is established. The neuralnetwork model trained again by the augmented training sample set hasbetter classification capabilities for each image type. Thereclassification result of the difficult-to-classify data and the datathat has been misclassified once by the newly trained neural networkmodel is more accurate, thereby reducing the workload of classificationreview.

In some embodiments, the determined data that has been misclassifiedtwice contains certain product defects that are difficult to berecognized, and the problem data may be quickly located. Furthermore,this technical problem may be solved from the algorithmic level or theservice level.

For example, from the algorithm level, it is possible to configureweights for the image type corresponding to the data that has beenmisclassified twice, so as to strengthen the weight of the loss value ofthis type during model training; and it is possible to perform dataenhancement (for example, flip, translation, splicing, and the like)operations on the image data of this image type) operations so as toincrease the data amount of the training sample for this image type.

For example, from the service level, the determination of whether theimage type (for example, the product defect type) corresponding to thedata that has been misclassified twice may be ignored. If it cannot beignored, it is possible to focus on collecting the sample data of theimage type to increase the data amount of the training sample of theimage type in the training sample set.

FIG. 4 shows a schematic view in some embodiments of the image dataclassification method according to the present disclosure.

As shown in FIG. 4, the marked small scale original training sample setis input into the neural network model (for example, the VGG 16convolutional neural model) as the data to be cleaned. Based on whetherthe membership probability of the classification result obtained by theneural network model is higher than the first threshold, theeasy-to-classify data set and the difficult-to-classify data set aredivided.

In some embodiments, the accuracy rate detection module may be used tocalculate the accuracy rate of each image type in the easy-to-classifydata set by random sampling. The low-accuracy data and thedifficult-to-classify data set are determined as the data to bereviewed; and the high-accuracy data is augmented to the originaltraining sample set.

By review processing of the classification accuracy, the image data thatis incorrectly classified is screened out from the low-accuracy data.The image data that is correctly classified is augmented to the originaltraining sample set; the image data that is incorrectly classified isdetermined as the data that has been misclassified once.

The augmented training sample set is used to train the neural networkmodel, and the updated neural network model is used to classify the datathat has been misclassified once and the difficult-to classify dataagain. The data that has been misclassified twice is obtained, and thecause of the classification error is analyzed based on the source of thedata that has been misclassified twice, so as to clean the image data.

In some embodiments, the neural network model is Visual Geometry GroupNetwork model.

In some embodiments, the original training sample set is obtained bycapturing product images during production process.

In some embodiments, the classification accuracy rate is aclassification accuracy rate of an image type in the easy-to-classify.

In the above-described embodiments, the workload in classificationreview of the image data is reduced, and the cleaning speed of the imagedata is accelerated. Moreover, the problems present in the data set arerapidly located, and the research and development progress of artificialintelligence projects is accelerated. It is possible to quantify theclassification accuracy of the data set, and effectively improve theclassification accuracy of the data set whilst, so that subsequentintelligent algorithms have more accurate and stable performance inactual scenarios.

FIG. 5 shows a block diagram in some embodiments of the image dataclassification device according to the present disclosure.

As shown in FIG. 5, the classification device 5 of the image datacomprises a classification unit 51, a determining unit 52, a processingunit 53 and a training unit 54.

The classification unit 51 inputs each image data into the neuralnetwork model for classification, and determines the image type to whicheach image data pertains and the membership probability.

The determining unit 52 divides the test image data with the membershipprobability greater than the first threshold into the easy-to-categorizedata set.

The processing unit 53 adds the test image data in the easy-to-classifydata set that has a classification accuracy rate less than or equal tothe second threshold and a correct classification result into thetraining sample set to generate an augmented training sample set.

The training unit 54 uses the augmented training sample set to train theneural network model so as to determine the image classification model.

In some embodiments, the processing unit 53 adds the test image datawith a classification accuracy rate greater than the second threshold inthe easy-to-classify data set to the training sample set so as togenerate an augmented training sample set.

In some embodiments, the classification unit 51 inputs each test imagedata into the neural network model trained by the augmented trainingsample set for classification; the processing unit 53 processes theaugmented training sample set based on whether the test image data thatis incorrectly classified this time pertains to the easy-to-classifydata set. The training sample set is augmented for processing; thetraining unit 54 uses the augmented training sample set after processingto train the neural network model again so as to update the imageclassification model.

In some embodiments, the processing unit 53 uses the test image datathat is correctly classified this time to augment the augmented trainingsample set again for training the neural network model again.

In some embodiments, the classification unit 51 inputs the first dataset and the second data set in each test image data into the trainedneural network model for classification. The first data set comprisestest image data in the easy-to-classify data set that has aclassification accuracy rate less than or equal to the second thresholdand an incorrect classification result. The second data set comprisestest image data pertaining to the difficult-to-classify data set. Thedifficult-to-classify data set is determined based on the image type towhich the test image data with the membership probability less than orequal to the first threshold pertains.

In some embodiments, in the case where the test image data that isincorrectly classified pertains to the first data set, the processingunit 53 determines the incorrectly marked samples required to befiltered in the augmented training sample set based on the test imagedata that is incorrectly classified.

In some embodiments, in the case where the test image data that isincorrectly classified pertains to the difficult-to-classify data set,the processing unit 53 obtains the sample data of the correspondingimage type based on the image type of the test image data that isincorrectly classified this time; the processing unit 53 augments theaugmented training sample set again. The difficult-to-classify data setis determined based on the image type to which the test image data withthe membership probability less than or equal to the first thresholdpertains.

In some embodiments, the processing unit 53 performs data enhancementprocessing on the test image data that is incorrectly classified thistime to obtain the sample data of the corresponding image type; theprocessing unit 53 augments the augmented training sample set again.

In some embodiments, the processing unit 53 calculates theclassification accuracy rate of the neural network model trained againby using the augmented training sample set after processing. Theprocessing unit 53 repeats at least one of the following steps until theclassification accuracy rate is greater than the third threshold: thetest image data in the easy-to-classify data set that has aclassification accuracy rate less than or equal to the second thresholdand a correct classification result is used to augment the trainingsamples; or the training samples are processed based on whether the testimage data that is incorrectly classified pertains to theeasy-to-classify data set.

In some embodiments, the processing unit 53 uses the linear adjustmentfactor and the exponential adjustment factor provided for the membershipprobability to process the focus loss function with the membershipprobability as a variable and determines the improved loss function fortraining the neural network model.

The linear adjustment factor and the exponential adjustment factor areconfigured such that: in the case where any membership probability isless than the membership probability of the intersection point of thefocal loss function and the improved loss function, the value of theimproved loss function corresponding to the membership probability isgreater than that of the focal loss function; in the case where anymembership probability is greater than that of the intersection point,the value of the improved loss function corresponding to the membershipprobability is less than that of the focal loss function.

In some embodiments, the improved loss function is determined accordingto (1−ŷ+ε)^(γ), where ŷ is the membership probability, ε is the linearadjustment factor, and γ is the exponential adjustment factor.

In some embodiments, the classification unit 51 inputs each image datainto the image classification model to determine the image type to whicheach image data pertains.

In some embodiments, each image data is the production line image dataof the production industry, and the image type is the product defecttype in the production line image data.

In some embodiments, the classification unit 51 inputs each image datainto an image classification model, and the processing unit marks eachimage data based on the classification result.

FIG. 6 shows a block diagram of other embodiments of the image dataclassification device according to the present disclosure.

As shown in FIG. 6, the image data classification device 6 in thisembodiment comprises a memory 61, and a processor 62 coupled to thememory 61, wherein the processor 62 is configured to implement the imagedata classification method according to any one of the embodiments ofthe present disclosure based on the instructions stored in the memory61.

Wherein, the memory 61 may comprise, for example, a system memory, afixed non-volatile storage medium, and the like. The system memory isstored with, for example, an operating system, an application program, aboot loader, a database, and other programs.

FIG. 7 shows a block diagram in still other embodiments of the imagedata classification device according to the present disclosure.

As shown in FIG. 7, the image data classification device 7 in thisembodiment comprises a memory 710, and a processor 720 coupled to thememory 710, wherein the processor 720 is configured to implement theimage data classification method according to any one of the embodimentsof the present disclosure based on the instructions stored in the memory710.

Wherein, the memory 710 may comprise, for example, a system memory, afixed non-volatile storage medium, and the like. The system memory isstored with, for example, an operating system, an application program, aboot loader, a database, and other programs.

The image data classification device 7 may also comprise an IN/OUTinterface 730, a network interface 740, a storage interface 750, and thelike. These interfaces 730, 740, 750, and the memory 710 and theprocessor 720 may be connected therebetween by a bus 760, for example.Wherein, the IN/OUT interface 730 provides a connection interface forinput and output devices such as a display, a mouse, a keyboard, a touchscreen, a microphone, and a speaker. The network interface 740 providesa connection interface for various networked devices. The storageinterface 750 provides a connection interface for external storagedevices such as SD card and U disk.

FIG. 8 shows a block diagram in some embodiments of the image dataclassification system of the present disclosure.

As shown in FIG. 8, the image data classification system comprises: animage data classification device 81 according to any one of theabove-described embodiments; and an image sensor 82 for obtaining theimage data. The classification device 81 is a hardware device such as ahardware processor or a hardware server that executes the image dataclassification method in any of the above embodiments.

Those skilled in the art will appreciate that the embodiments of thepresent disclosure may be provided as a method, system, or computerprogram product. Accordingly, the present disclosure may take the formof an entirely hardware embodiment, an entirely software embodiment, ora combination of software and hardware aspects. Moreover, the presentdisclosure may take the form of a computer program product embodied inone or more computer-usable non-transitory storage media (comprising butnot limited to disk memory, CD-ROM, optical memory, and the like)comprising computer usable program codes therein.

So far, the image data classification method, the image dataclassification device, the image data classification system, and thenon-volatile computer-readable storage medium according to the presentdisclosure have been described in detail. Some details well known in theart are not described in order to avoid obscuring the concept of thepresent disclosure.

According to the above description, those skilled in the art would fullyunderstand how to implement the technical solutions disclosed here.

The method and system of the present disclosure may be implemented inmany manners. For example, the method and system of the presentdisclosure may be implemented by software, hardware, firmware, or anycombination of software, hardware, and firmware. The above-describedsequence for the steps of the method is merely for illustrativepurposes, and the steps of the method according to the presentdisclosure are not limited to the sequence specifically described aboveunless otherwise specified. Moreover, in some embodiments, the presentdisclosure may also be embodied as programs recorded in a recordingmedium, which comprise machine readable instructions for implementingthe method according to the present disclosure. Thus, the presentdisclosure also covers a recording medium that stores programs forperforming the method according to the present disclosure.

Although some specific embodiments of the present disclosure have beendescribed in detail by way of examples, those skilled in the art shouldunderstand that the above examples are only for the purpose ofillustration and are not intended to limit the scope of the presentdisclosure. It should be understood by those skilled in the art thatmodifications to the above embodiments may be made without departingfrom the scope and spirit of the present disclosure. The scope of thepresent disclosure is defined by the appended claims.

What is claimed is:
 1. An image data classification method, comprising:inputting test image data into a neural network model trained by usingan original training sample set for classification, and determining animage type to which the test image data belongs and a membershipprobability of the image data belonging to the image type; establishingan easy-to-classify data set, according to test image data with amembership probability greater than a first threshold; adding test imagedata in the easy-to-classify data set that has a classification accuracyrate less than or equal to a second threshold and a correctclassification result to the original training sample set to generate anaugmented training sample set; and determining an image classificationmodel by using the augmented training sample set to train the neuralnetwork model.
 2. The image data classification method according toclaim 1, further comprising: adding test image data with aclassification accuracy rate greater than the second threshold in theeasy-to-classify data set to the training sample set so as to generatethe augmented training sample set.
 3. The image data classificationmethod according to claim 1, further comprising: inputting the testimage data into the neural network model trained by the augmentedtraining sample set for classification; processing the augmentedtraining sample set based on whether test image data that is incorrectlyclassified this time pertains to the easy-to-classify data set; andupdating the image classification model by using the augmented trainingsample set after processing to train the neural network model again. 4.The image data classification method according to claim 3, furthercomprising: using test image data that is correctly classified this timeto augment the augmented training sample set again so as to train theneural network model again.
 5. The image data classification methodaccording to claim 3, wherein the inputting the test image data into theneural network model trained by the augmented training sample set forclassification comprises: inputting a first data set and a second dataset of the test image data into the neural network model trained by theaugmented training sample set for classification, wherein the first dataset comprises test image data in the easy-to-classify data set that hasa classification accuracy rate less than or equal to the secondthreshold and an incorrect classification result, and the second dataset comprises test image data in a difficult-to-classify data set,wherein the difficult-to-classify data set is determined based on animage type to which test image data with a membership probability lessthan or equal to the first threshold belongs.
 6. The image dataclassification method according to claim 5, wherein the processing theaugmented training sample set based on whether test image data that isincorrectly classified this time pertains to the easy-to-classify dataset comprises: determining an incorrectly marked samples required to befiltered in the augmented training sample set based on the test imagedata that is incorrectly classified in a case that the test image datathat is incorrectly classified belongs to the first data set.
 7. Theimage data classification method according to claim 3, wherein theprocessing the augmented training sample set based on whether test imagedata that is incorrectly classified this time pertains to theeasy-to-classify data set comprises: obtaining sample data of an imagetype of the test image data that is incorrectly classified this time toaugment the augmented training sample set again, in the case where thetest image data that is incorrectly classified pertains to adifficult-to-classify data set, wherein the difficult-to-classify dataset is determined based on an image type to which test image data with amembership probability less than or equal to the first thresholdbelongs.
 8. The image data classification method according to claim 7,wherein the obtaining sample data of an image type of the test imagedata that is incorrectly classified this time to augment the augmentedtraining sample set again comprises: performing data enhancementprocessing on the test image data that is incorrectly classified thistime, obtaining the sample data of the image type of the test image datathat is incorrectly classified this time, and augmenting the augmentedtraining sample set again.
 9. The image data classification methodaccording to claim 3, further comprising: calculating a classificationaccuracy rate of the neural network model trained again by using theaugmented training sample set after processing; repeating at least oneof the following steps until the classification accuracy rate of theneural network model is greater than a third threshold in a case wherethe classification accuracy rate of the neural network model is lessthan or equal to the third threshold: augmenting test image data in theeasy-to-classify data set that has a classification accuracy less thanor equal to the second threshold and a correct classification result tothe training sample set, and training the neural network model by usingthe training sample set augmented; or processing the training sample setbased on whether the test image data that is incorrectly classified thistime belongs to the easy-to-classify data set and training the neuralnetwork model using the processed training sample set.
 10. The imagedata classification method according to claim 1, further comprising:using a linear adjustment factor and an exponential adjustment factorprovided for a membership probability to process a focus loss functionwith the membership probability as a variable, and determining animproved loss function to train the neural network model, wherein thelinear adjustment factor and the exponential adjustment factor areconfigured such that value of the improved loss function correspondingto the membership probability is greater than that of the focal lossfunction in a case where any membership probability is less than amembership probability of the intersection point of the focal lossfunction and the improved loss function, and the value of the improvedloss function corresponding to the membership probability is less thanthat of the focus loss function in a case where any membershipprobability is greater than a membership probability of the intersectionpoint.
 11. The image data classification method according to claim 10,wherein, the improved loss function is determined according to(1−ŷ+ε)^(γ), wherein ŷ is the membership probability, ε is the linearadjustment factor, and γ is the exponential adjustment factor.
 12. Theimage data classification method according to claim 1, furthercomprising: inputting image data into the image classification model anddetermining an image type to which the test image data belongs.
 13. Theimage data classification method according to claim 12, wherein theimage data is production line image data of production industry, and theimage type is a product defect type of the production line image data.14. The image data classification method according to claim 1, furthercomprising: inputting image data into the image classification model andmarking the image data based on a classification result.
 15. The imagedata classification method according to claim 1, wherein the neuralnetwork model is Visual Geometry Group Network model.
 16. The image dataclassification method according to claim 1, wherein the originaltraining sample set is obtained by capturing product images duringproduction process.
 17. The image data classification method accordingto claim 1, wherein the classification accuracy rate is a classificationaccuracy rate of an image type in the easy-to-classify data setcalculated by using multiple accuracy rate detection modules.
 18. Animage data classification device, comprising: a memory; and a processorcoupled to the memory, wherein the processor is configured to implementthe image data classification method according to claim 1 based oninstructions stored in the memory.
 19. An image data classificationsystem, comprising: an image data classification device according toclaim 18; and an image sensor for obtaining image data.
 20. Anon-volatile computer-readable storage medium having a computer programstored thereon, which when executed by a processor implements the imagedata classification method according to claim 1.